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Abstract. This contribution reviews the parallel dynamics of Q-Ising neural 
networks for various architectures: extremely diluted asymmetric, layered feedforward, 
extremely diluted symmetric, and fully connected. Using a probabilistic signal-to- 
noise ratio analysis, taking into account all feedback correlations, which are strongly 
dependent upon these architectures the evolution of the distribution of the local field 
is found. This leads to a recursive scheme determining the complete time evolution 
of the order parameters of the network. Arbitrary Q and mainly zero temperature 
are considered. For the asymmetrically diluted and the layered feedforward network a 
closed-form solution is obtained while for the symmetrically diluted and fully connected 
architecture the feedback correlations prevent such a closed-form solution. For these 
symmetric networks equilibrium fixed-point equations can be derived under certain 
conditions on the noise in the system. They are the same as those obtained in a 
thermodynamic replica-symmetric mean-field theory approach. 



1. Introduction 

Artificial neural networks have been widely applied to memorize and retrieve 
information. During the last few years there has been considerable interest in neural 
networks with multistate neurons (see |l] and references cited therein) in order to 
function as associative memories for gray-toned or coloured patterns or to allow for 
a more complicated internal structure in the retrieval, e.g., a distinction between 
background and patterns. 

Here we review the dynamics of so-called Q-Ising neural networks (see the references 
in [Q) for arbitrary Q. They are built from Q-Ising spin-glasses |3], f|] with couplings 
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defined in terms of patterns through a learning rule. For Q = 2 one finds back the 
Hopfield model |5|, |6|], for Q — > oo one has an analog network (see |7| and references 
therein). One of the aims of these networks is to memorize a number of patterns and 
find them back as attractors of the retrieval process. Consequently these networks are 
also interesting from the point of view of dynamical systems. 

Besides a learning rule one also needs to specify an architecture indicating how 
the spins (=neurons) are connected with each other. Several architectures have been 
studied in the literature for different purposes. From a practical point of view mostly 
perceptrons or, more general, feedforward layered networks are used since a very long 
time (see, e.g., || for a history). Hopfield 0, |6| studied a fully connected network 
with symmetric couplings because it satisfies the detailed balance principle and hence a 
Hamiltonian can be defined. Asymmetrically diluted models were used because their 
dynamics can be solved exactly and because they can learn us something about the loss 
of information content when some of the synaptic couplings break down. 

In this contribution we review the study of the parallel dynamics of these types of 
network using a probabilistic approach (see, e.g., [[U], [□]]). In more detail, employing a 
signal-to-noise ratio analysis based on the law of large numbers (LLN) and the central 
limit theorem (CLT) we derive the evolution of the distribution of the local field at every 
time step. This allows us to obtain a recursive scheme for the evolution of the relevant 
order parameters in the system being, in general, the main overlap for the condensed 
pattern, the mean of the neuron activities and the variance of the residual overlap 
responsible for the intrinsic noise in the dynamics of the main overlap (sometimes called 
the width parameter). The details of this approach depend in an essential way on the 
architecture because different temporal correlations are possible. 

For extremely diluted asymmetric and layered feedforward architectures recursion 
relations have been obtained in closed form directly for the relevant order parameters 



|9|. [L2|, [13|, |14| . This has been possible because in these types of networks there are 
no feedback correlations as time progresses. As a technical consequence the local field 
contains only Gaussian noise leading to an explicit solution. 

For the parallel dynamics of networks with symmetric connections, however, things 
are quite different |], [H], [11]. Even for extremely diluted versions of these systems 
jl5l , [16| , |I"7H feedback correlations become essential from the second time step onwards, 
complicating the dynamics in a nontrivial way. Therefore, explicit results concerning the 
time evolution of the order parameters for these models have to be obtained indirectly 
by starting from the distribution of the local field. Technically speaking, both for the 
symmetrically diluted and fully connected architectures the local field contains both a 
discrete and a normally distributed part. The difference between the diluted and fully 
connected models is that the discrete part at a certain time t does not involve the spins 
at all previous times t — 1, t — 2, . . . up to but only the spins at time step t — 1. 
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But in both cases the discrete part prevents a closed-form solution of the dynamics for 
the relevant order parameters. Nevertheless, the development of a recursive scheme is 
possible in order to calculate their complete time evolution. In this way a comparative 
discussion of the parallel dynamics at zero temperature for the various architectures 
specified above is possible. 

Finally, by requiring the local field to become time-independent implying that some 
correlations between its Gaussian and discrete noise parts are neglected we can obtain 
fixed-point equations for the order parameters. It turns out that they are equivalent to 
the fixed-point equations obtained through a thermodynamic replica-symmetric mean- 
field theory approach. 

At this point we remark that we do not aim for complete rigour in our derivations. 
From the point of view of rigorous mathematics, the Hopfield model and, in general, 
spin-glass theory is recognized to be an extremely difficult, if not imposible, field. For 
a recent overview of the modest results obtained, mostly concerning thermodynamics, 
we refer to 



The rest of this contribution is organized as follows. In Section we introduce 
the model, its dynamics and the Hamming distance as a macroscopic measure for the 
retrieval quality. In Section |3] we use the probabilistic approach in order to derive 
a recursive scheme for the evolution of the distribution of the local field, leading 
to recursion relations for the order parameters. The differences between the various 
architectures are outlined. We do not aim for complete rigour and mostly concentrate 
on zero temperature. In Section [| we discuss the evolution of the system to fixed-point 
attractors. Some concluding remarks are given in Section B. 



2. Q-Ising neural networks 

Consider a neural network A consisting of N neurons which can take values <7j from a 
discrete set S = { — 1 = si < s 2 < ... < sq = +1}. The p patterns to be stored in 
this network are supposed to be a collection of independent and identically distributed 
random variables (i.i.d.r.v.), {£f G S}, \i G V = {1, . . . ,p} and i G A, with zero mean, 
E[£i] = 0, and variance A = Var[£f]. The latter is a measure for the activity of the 
patterns. We remark that for simplicity we have taken the patterns and the neurons out 
of the same set of variables but this is no essential restriction. Given the configuration 
cr A (t) = {<Jj(t)},j G A = {1, . . . , N}, the local field in neuron i equals 

k(«rA(f)) = £ J <*(*to(*) (1) 

with Jij the synaptic coupling from neuron j to neuron i. In the sequel we write the 
shorthand notation h\j(t) = hi((J\(t)). 
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It is clear that the J^- explicitly depend on the architecture. For the extremely 
diluted (ED), both symmetric (SED) and asymmetric (AED), and the fully connected 
(FC) architectures the couplings are time-independent and the diagonal terms are 
absent, i.e. Ju = 0. The configuration cr A (t = 0) is chosen as input. For the layered 
feedforward (LF) model the time dependence of the couplings is relevant because the set- 
up of the model is somewhat different. There each neuron in layer t is unidirectionally 
connected to all neurons on layer t + 1 and Jij(t) is the strength of the coupling from 
neuron j on layer t to neuron i on layer t + 1. The state cr A (t + 1) of layer t + 1 is 
determined by the state cr a(^) of the previous layer t. 

In all cases the couplings are chosen according to the Hebb rule such that we can 
write 

4 D =^Eere? for ^j, (2) 

4° =m?:&t! for MJ, (3) 

Jg F (*) = ^EW + 1 )W, (4) 

with the {cy = 0,1}, i,j G A chosen to be i.i.d.r.v. with distribution Pr{cjj = x} = 
(1 — C/N)S x fi + {C/N)6 x> i and satisfying c^- = c^, Cu = for symmetric dilution, and 
Cij and Cji statistically independent (with ca = 0) for asymmetric dilution. 

At zero temperature all neurons are updated in parallel according to the rule 

<Ti(t) -> (7i(t + 1) = s k : minei[s|cr A (t)] = ej[s fc |cr A (t)] . (5) 

We remark that this rule is the zero temperature limit T = (3~ l — > of the stochastic 
parallel spin-flip dynamics defined by the transition probabilities 

d; /, ±1 x _ q\ /.x, exp[-/3ei(s & |<r A (t))] , , 

E s6 sexp[-/3ei(s|<T A (i))] 

Here the energy potential ej[s|cr A ] is defined by [0 

ei[s\o- A ] = -Uhi{(T K )s - bs 2 ] , (7) 

where 6 > is the gain parameter of the system. The updating rule (|5|) is equivalent to 
using a gain function g 6 (-), 

<n(t + 1) = g b (h A ,i(t)) 
Q 

Sbi x ) = J2 s k[° [K s k+i +s k )-x]—0 [b(s k + s A _i) - x)) (8) 
fc=l 

with s = — oo and sq + i = +oo. For finite Q, this gain function g b (-) is a step function. 
The gain parameter b controls the average slope of g 6 (-). 



In order to measure the retrieval quality of the system one can use the Hamming 
distance between a stored pattern and the microscopic state of the network 

d(e(t),<T A (t)) = ^Y,W(t)-*i(t)] 2 - (9) 

This introduces the main overlap and the arithmetic mean of the neuron activities 

<(*) = 4rEer(*)^(*), ^P; a A {t) = ±Y>M 2 ■ ( 10 ) 

«eA «eA 

We remark that for Q = 2 the variance of the patterns A — 1, and the neuron activity 
a A (£) = 1. 



3. Solving the dynamics 

3.1. Correlations 

We first discuss some of the geometric properties of the various architectures which are 
particularly relevant for the understanding of their long-time dynamic behaviour. 

For a fully connected architecture there are two main sources of strong correlations 
between the neurons complicating the dynamical evolution : feedback loops and the 
common ancestor problem P0| |. Feedback loops occur when in the course of the time 



evolution, e.g., the following string of connections is possible: i — > j —>■ k —>■ i. We 
remark that architectures with symmetric connections always have these feedback loops. 
In the absence of these loops the network functions in fact as a layered system, i.e., only 
feedforward connections are possible. But in this layered architecture common ancestors 
are still present when, e.g., for the sites i and j there are sites in the foregoing time 
steps that have a connection with both i and j. 

In extremely diluted asymmetric architecture these sources of correlations are 
absent. This class of neural networks was introduced in connection with Q = 2-Ising 
models ||. We recall that the couplings are then given by eq. (2) and that in the limit 
N — > oo two important properties of this network are essential fl, |2lfl. The first property 
is the high asymmetry of the connections, viz. 



Pr{ Cij = Cji } = ( , Pr{ % = lA^ = 0} = ^(l-^). (11) 



Therefore, the number of symmetric connections in the infinite configuration c = 
{cij},i,j / i G N is finite with probability one, i.e. almost all connections of the 
graph Gn( c ) — {{h j) '■ c ij = M,i / « 6 N} are directed : c.y ^ Cji. 

The second property in the limit of extreme dilution is the directed local Cayley-tree 
structure of the graph Gn(c). By the arguments above the probability (c) that k 



6 



connections are directed towards a given site i G A is 

Ff> (C) . Pr{ * = m) = (!-%)"-* (12) 



k\(N — k)\ \NJ V N 

where = {cji = 1, j G A \ i} is the in-tree for i and \T { | its cardinality. This 
probability is equal to Pr{fc = \T^ outS> \ = \{cij — l,j G A \ i}\} for connections directed 
outward a given site i G A. In the limit of extreme dilution we get a Poisson distribution 

lim F<; A \C) = ^e~ c . (13) 

Hence, the mean value of the number of in (out) connections for any site i G A is 
^j-|^( in )(°"*)|j _ q fj^g probability that two sites i and i 1 have site j as a common 
ancestor is obviously equal to C/N. From ^[It/ 1 ™^] = C it follows that after t time 
steps the cardinality of the cluster of ancestors for site i will be of the order of C*. The 
same is valid for site i' . Therefore, the probability that the sites i and i' have disjoint 
clusters of ancestors approaches (1 — C* /N) ct ~ exp(— C 2t /N) for N ^> 1. 

So we find that in the limit of extreme dilution : (i) Almost all (i.e. with probability 
1) feedback loops in G*n( c ) are eliminated, (ii) With probability 1 any finite number 
of neurons have disjoint clusters of ancestors. So we first dilute the system by taking 
N oo and then we take the limit C — > oo in order to get infinite average connectivity 
allowing to store infinitely many patterns p. 

This implies that for this asymmetrically diluted model at any given time step t 
all spins are uncorrelated and, hence, the first step dynamics describes the full time 
evolution of the network. 

For the symmetrically diluted model the architecture is still a local Cayley-tree 
but no longer directed and in the limit — >• oo the probability that the number of 
connections Tj = {j G A|qj = 1} giving information to the the site j 6 A is still a 
Poisson distribution with mean C = E[\Ti\]. However, at time t the spins are no longer 
uncorrelated causing a feedback from t > 2 onwards |H] . 

In order to solve the dynamics we start with a discussion of the first time step 
dynamics, the form of which is independent of the architecture. 



3.2. First time step 

Consider a fully connected network. Suppose that the initial configuration of the 
network {<Xj(0)}, i G A, is a collection of i.i.d.r.v. with mean E[<7j(0)] = 0, variance 
Var[(Xj(0)] = do, and correlated with only one stored pattern, say the first one {^}: 



(14) 
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This pattern is said to be condensed. By the law of large numbers (LLN) one gets for 
the main overlap and the activity at t = 



m l (0) = lim m»(0) 

N— >oo 



a(0) 



lim oa(0) 

N— >oo 



I E[^V,(0)] = 
Ek 2 (0)]=a 



(15) 
(16) 



where the convergence is in probability [^2|. In order to obtain the configuration at 
t — 1 we first have to calculate the local field (JJ) at t — 0. To do this we employ 
the signal-to-noise ratio analysis (see, e.g.,|T0], |TJ]). Recalling the learning rule (3) we 
separate the part containing the condensed pattern, i.e., the signal, from the rest, i.e., 
the noise to arrive at 



E ?>j(0)(17) 

jeA\i v ^g-p\i vj"i jeA\i 

where a = p/N. The properties of the initial configurations (|H])-(16) assure us that the 



summation in the first term on the r.h.s of ( |17"D converges in the limit N 
lim 1TT7 E £f*i(0) =m l (Q). 



oo to 



jeA\i 



The first term Qm (0) is independent of the second term on the r.h.s of (|TTD. This 
second term contains the influence of the non-condensed patterns causing the intrinsic 
noise in the dynamics of the main overlap. In view of this we define the residual overlap 



lim ?"a(£) = lim = 



Ee^iW »eV\{l}. 



(19) 



Applying the CLT to this second term in (|TTD we find 



lim 

N^oo 



f E « Ni (o) 



lim 



2? 



i= E £ 

aJV(0, AD(0)) 



'^5" (0) (20) 

(21) 



where the quantity A/"(0, V) represents a Gaussian random variable with mean and 
variance V and where -D(O) = Varfr^O)] = a(0) Thus we see that in fact the variance of 
this residual overlap, i.e., D(t) is the relevant quantity characterising the intrinsic noise. 

In conclusion, in the limit iV — > oo the local field is the sum of two independent 
random variables, i.e. 



hi(0) = lim h Ai (0) = Ci^fO) + y/aAf(0,a(0)). 



(22) 



For a more rigourous discussion of the first time step for the underlying spin-glass model 
we refer to p3j. At this point we note that the structure (1221) of the distribution of the 
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local field at time zero - signal plus Gaussian noise - is typical for all architectures 
discussed here because the correlations caused by the dynamics only appear for t > 1. 
Some technical details are different for the various architectures. The first change in 
details that has to be made is an adaptation of the sum over the sites j to A for the 
layered feedforward architecture and to Tj, the part of the tree connected to neuron i, 
in the diluted architectures. The second change is that for the diluted architectures 
an additional limit C — > oo has to be taken besides the N — > oo limit. So in the 
thermodynamic limit C, N — > oo all averages will have to be taken over the treelike 
structure, viz. J2i^A ~^ ^YlieTj- Furthermore a = p/N has to be replaced by 
a = p/C. 



3.3. Recursive dynamical scheme 

The key question is then how these quantities evolve in time under the parallel dynamics 
specified before. For a general time step we find from the LLN in the limit C, N — > oo 



for the main overlap and the activity (Hp 



m\t + l) £ i(&V<(*+ !)>/>>>, a{t+l) P = (23) 
with the thermal average defined as 

»( ,+ 1 >»' = ¥tiS¥^ (24) 

£<xes exp^/? <r{hi(t) - ba)\ 

where hi(t) = limjy^oo h^it). In the above ((■)) denotes the average both over the 
distribution of the embedded patterns {£f } and the initial configurations {crj(O)}. The 
average over the latter is hidden in an average over the local field through the updating 
rule (8). In the sequel we focus on zero temperature. Then, using eq. (8) these formula 
reduce to 

m\t + l) P ^ jUhMt)))), <t + l) ^ (i&W)))) ■ (25) 

As seen already in the first time step, we have to study carefully the influence of the 
non-condensed patterns causing the intrinsic noise in the dynamics of the main overlap. 
The method used to obtain these order parameters is then to calculate the distribution 
of the local field as a function of time. In order to determine the structure of the local 
field we have to concentrate on the evolution of the residual overlap. The details of this 
calculation are very technical and depend on the precise correlations in the system and 
hence on the architecture of the network as discussed before. For these technical details 
we refer to the relevant literature P], 0, [12], |14], |15[ . Here we give an extensive discussion 



of the results obtained. 

In general, the distribution of the local field at time t + 1 is given by 

hiit+l) =&m\t+l)+N(0,aa(t+l))+ X (t)[F(h t (t)-&m\t))+Baa l (t)}(2Q) 



where F and B are binary coefficients given below, which depend on the specific 
architecture. From this it is clear that the local field at time t consists out of a discrete 
part and a normally distributed part, viz. 

hi(t) = Mi(t)+M(p,V(t)) (27) 
where Mi(t) satisfies the recursion relation 

Mi(t + 1) = X (t)[F{Mi(t) - &m\t)) + Ba<Ti(t)] + i]m l (t + 1) (28) 
and where V(t) = a AD it) with D(t) itself given by the recursion relation 



D(t + 1) = + L x \t)D{t) + 2F X (t)Cav[r»(t), r"(f)] 

where L is again a coefficient specified below. The quantity x(t) reads 

O-i 

X(t) = /*?(*) ( 6 ( s *+i + s k ))(s k +i - s k ) 



k=l 



where is the probability density of hf(t) = limA^oo h ki (t) with 



Furthermore, r^(t) is defined as 
P(£) = lim 



AT->oo AVN 



(29) 



(30) 



(31) 



(32) 



Finally, as can be read off from eq. (|28f) the quantity Mi(t) consists out of the signal 
term and a discrete noise term, viz. 



t-2 



t-1 



ffi(0- (33) 



M 4 (t) = &m\t) + B x oc X {t - l)<7i(t - 1) + £? 2 £ « 

t'=0 

Since different architectures contain different correlations not all terms in these final 
equations are present. In particular we have for the coefficients F, B, L, B\ and B 2 
introduced above 

F B L B\ B 2 



FC 
SED 
LF 
AED 



11111 
10 10 
10 




(34) 



with B indicating the feedback caused by the symmetry in the architectures and L the 
common ancestors contribution. 

At this point we remark that in the so-called theory of statistical neurodynamics 
2~3| , one starts from an approximate local field by leaving out any discrete noise (the 
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term in o"j(t)). As a consequence the covariance in the recursion relation for D(t) can 
be written down more explicitly since only Gaussian noise is involved. For more details 



we refer to P6 



We still have to determine the probability density of /^m in eq. Q3HD, which in 
the thermodynamic limit equals the probability density of fhdt)- This can be done by 
looking at the form of Mj(t) given by eq. The evolution equation tells us that 



(Ji(t') can be replaced by g b (hi(t' — 1)) such that the second and third terms of Mj(t) 
are the sums of stepfunctions of correlated variables. These are also correlated through 
the dynamics with the normally distributed part of hi(t). Therefore the local field can 
be considered as a transformation of a set of correlated normally distributed variables 
x s , s = 0, . . . ,t — 2,t, which we choose to normalize. Defining the correlation matrix 
W = (p(s, s') = E[x a x s >]) we arrive at the following expression for the probability density 
of the local field at time t 

fhi(t)(y) = I II dx * dx t s(y- Mi(t) - sj a AD (t) x t 

J s=0 ^ 

1 t 1 «r-l_T 



x - exp — xWV (35) 

/det(27rW0 V 2 



with x = (xq, . . . x t -2,x t ). For the symmetrically diluted case this expression simplifies 
to 

W2] 

fhi(t)(y) = / II dx t-2s $ [y-£iTn (t) - ax{t)(Ji{t) - sjaa{t)x t 
1 / 1 „_! -r 



x ; exp — xW~W (36) 

/det(27TW) 



with x = ({x s }) = (x t -2[t/2], ■ ■ -x t -2,Xt). The brackets [t/2] denote the integer part of 
t/2. 

So the local field at time t consists out of a signal term, a discrete noise part and 
a normally distributed noise part. Furthermore, the discrete noise and the normally 
distributed noise are correlated and this prohibits us to derive a closed expression for 
the overlap and activity. 

Together with the eqs. (25) for m 1 (t + 1) and a(t + 1) the results above form a 
recursive scheme in order to obtain the order parameters of the system. The practical 
difficulty which remains is the explicit calculation of the correlations in the network at 
different time steps as present in eq. (p9|). 

For AED and LF architectures this scheme leads to an explicit form for the recursion 
relations for the order parameters 

m^t + l) = 5^j(^e(t+l) J Dzg(e(t + l)m\t) + ^aAD(t)z)^ (37) 



a(t +1) = (^J Dzg 2 {i\t + l)m\t) + sfaAD(t) z) 



D{t+1) 



a(t 



±11 + A 

A aA 



J Dz zg(^(t + l)m\t) + s]aAD{t) z) 
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(3? 



1 2 



(39) 



For the AED architecture (L = 0) the second term on the r.h.s. of (39) coming from 
the correlations caused by the common ancestors is absent. For the LF architecture we 
remark that this explicit solution requires an independent choice of the representations of 
the patterns at different layers. At finite temperatures analogous recursion relations for 



the AED and LF networks can be derived [12, 14] by introducing auxiliary thermal fields 



27J in order to express the stochastic dynamics within the gain function formulation 
of the deterministic dynamics. Furthermore, damage spreading B 28, 29], i.e., the 



evolution of two network configurations which are initially close in Hamming distance 
can be studied 



12, 14 



Finally, a complete self-control mechanism can be built in 
the dynamics of these systems by introducing a time- dependent threshold in the gain 
function improving, e.g., the basins of attraction of the memorized patterns [|30|j ~ [|32f . 

For explicit examples of this dynamical scheme with numerical results we refer to 
P|, [TS|]. By using the recursion relations the first few time steps are written out explicitly 
and studied numerically, e.g, for the Q = 2 and Q = 3 FC and SED models with 
equidistant states and a uniform distribution of patterns. These results are compared 
with the approximations studied in the literature ]T§ [TTj, [R| |T7|, |2§ by 
neglecting some feedback correlations for t > 2. In the whole retrieval region of these 
networks we find that the first four or five time steps give us already a clear picture of 
their time evolution. 



4. Fixed-point equations 

Equilibrium results for the AED and LF Q-Ising models are obtained immediately by 
straightforwardly leaving out the time dependence in (37)- (39) (see |L2]],Jl4|]), since the 



evolution equations for the local field and the order parameters do not change their 
form as time progresses (see eqs. (37)- (39)). This still allows small fluctuations in the 
configurations {o"i}. The difference between the fixed-point equations for these two 
architectures is that for the AED model the variance of the residual noise, D(t) is 
simply proportional to the activity of the neurons at time t while for the LF model a 
recursion is needed. 

For the SED and FC architectures, however, the evolution equations for the order 
parameters do change their form by the explicit appearance of the {cr^t')}, t' — 1, . . . , t 
term. Hence we can not use the simple procedure above to obtain the fixed-point 
equations. Instead we derive the equilibrium results of our dynamical scheme by 
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requiring through the recursion relations (|26| ) that the distribution of the local field 
becomes time- independent. This is an approximation because fluctuations in the 
network configuration are no longer allowed. In fact, it means that out of the discrete 
part of this distribution, i.e., Mj(t) (recall fl33|)), only the a,i{t — 1) term is kept besides, 
of course, the signal term. This procedure implies that the main overlap and activity 
in the fixed-point are found from the definitions ([10]) and not from leaving out the time 
dependence in the recursion relation (25). 

We start by eliminating the time-dependence in the evolution equations for the local 
field (p6|). This leads to 

hi = £V + [rTW(0, aa) + [xT Wi (40) 

with x ar = 1 — Fx being 1 for the SED and 1 — x f°r the FC model and hi = 
lim^oo h{(t). This expression consists out of two parts: a normally distributed part 
hi = AflQm 1 , aa/[x ar ] 2 ) and some discrete noise part. At this point some remarks are 
in order. First, the discrete noise coming from the correlations of the {(?i(t)} at different 
time steps (here only the preceding time step is considered) is inherent in the SED and 
FC dynamics. Second, the so-called self-consistent signal-to-noise ratio analysis of the 
FC network considered in the literature |39|, |40| starts from such a type of equation by 
assuming the presence of a term proportional to the output in the local field without 
any reference or argumentation based upon the underlying dynamics of the network. 
Employing this expression in the updating rule (8) one finds 

*i = g 6 & + [x° r ]"Wi) (41) 



where hi = M^m 1 ,aa) is the normally distributed part of eq. (pEUp. This is a self- 
consistent equation in <7j which in general admits more than one solution. These types 
of equation have been solved in the literature in the context of thermodynamics using 
a geometric Maxwell construction [ 39fl , [fl0|l . We remark that for analog networks the 



geometric Maxwell construction is not necessary: the fixed-point equation ( ffl|) has only 



one solution. For more technical details we refer to |p6 
This approach leads to a unique solution 

a t = g- b {hi), b = b-[2x a TW- (42) 



We remark that plugging this result into the local field fl40D tells us that the latter is 
the sum of two Gaussians with shifted mean (see also [[37j). 

Using the definition of the main overlap and activity (|ToD in the limit iV — > oo for 
the FC model and limit C, N —>■ oo for the SED model, one finds in the fixed point 



i 1 J Vz g s (£W + V^AD z) \\ (43) 
Vz gf (£W + V a AD z) \\ . (44) 
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From (|29| ) and (|30]) one furthermore sees that 

d = [rr 2 «A4 

with 

J Vz zgi (i l m l + VaAD. 



1 



X 



VaAD 



(45) 



(46) 



These resulting equations (43)- fl45|) are the same as the fixed-point equations derived 
from a replica-symmetric mean-field theory treatment in [ 41 ^ [ 44 ] . Their solution leads 



to capacity-gain parameter phase diagrams (see, e.g.,|44 



5. Concluding remarks 

An evolution equation is derived for the distribution of the local field governing the 
parallel dynamics at zero temperature of extremely diluted symmetric and asymmetric, 
layered feedforward and fully connected Q > 2-Ising networks. All feedback correlations 
are taken into account. In general, this distribution is not normally distributed but 
contains a discrete noise part. 

Employing this evolution equation a general recursive scheme is developed allowing 
one to calculate the relevant order parameters of the system, i.e., the main overlap, the 
activity and the variance of the residual noise for any time step. For the extremely 
diluted asymmetric and the layered feedforward architectures this scheme immediately 
leads to explicit recursion relations for the order parameters because the discrete noise 
part in the local field is absent. For the extremely diluted and the fully connected 
architectures equilibrium fixed-point equations for the order parameters are obtained 
under the condition that the local field becomes time- independent, meaning that some 
of the discrete noise is neglected. The resulting equations are the same as those derived 
from a replica-symmetric mean-field theory approach. 
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